Usability and Performance Testing of Pointcheck™: A Noninvasive Neutropenia Screening Device for Chemotherapy Outpatients

Introduction

Every year, approximately 110,000 cancer patients treated with cytotoxic chemotherapy in the US endure at least one episode of febrile neutropenia (FN). Each FN episode will require an approximate 8-9-day admission costing $25,000 and may have associated mortality rates as high as 7% . Timely detection and awareness of severe neutropenia (i.e., Absolute Neutrophil Count (ANC) <500/µL) is key to managing FN. The current gold standard relies on blood draws and analysis is hence limited to the clinical setting. We have developed PointCheck™, a noninvasive, portable technology (Fig. 1-A) that can automatically monitor for severe neutropenia and enables prompt detection in the home- as described in Bourquard et al (2018) and Pablo-Trinidad et al (2019).

Methods

We conducted a multicenter study to evaluate the usability and diagnostic performance of PointCheck™ in a cohort of 26 participants. The primary study objectives were to assess device usability when operated by first-time users and to meet a priori specified diagnostic performance goals in achieving 92% sensitivity (true positive rate) and 80% specificity (true negative rate).

Study coordinators read a short script providing the participants with information about how to use the device. In addition, participants watched a 1-minute tutorial video, were provided with a one-page user guide before attempting to take a measurement on their own following the on-screen Graphical User Interface instructions (Fig. 1-B). They did so autonomously to simulate home usage. Usability data was collected using a combination of methods, including the think-aloud technique, a short-answer qualitative questionnaire, and a standardized quantitative System Usability Survey (SUS)- a usability evaluation method widely used in UI/UX design comprising 10 questions. Participants were also given the opportunity to document their thoughts, feelings, and experience using the device in the form of an e-questionnaire containing 4 questions about their perceptions. Design and functionality issues were primarily identified by the research staff through observation of participants and documentation of missed or incorrectly completed steps.

Participants also had their blood drawn for a comparison with their complete blood count (CBC) to assess diagnostic performance. PointCheck™ measurement results were compared with clinical standard CBC tests and assessed for accuracy in classifying subjects as severely neutropenic (<500/µL, grade IV neutropenia) or non-severely neutropenic (≥500/µL).

Results

Usability and diagnostic performance data was gathered from untrained cancer patients and healthy volunteers. We included patients with lymphoma and myeloma, among other tumor types, who visited either Boston Medical Center (BMC) and Hospital Universitario 12 de Octubre (H12O) for routine chemotherapy administration. The healthy volunteers were recruited at the Massachusetts Institute of Technology Clinical Research Center (MITCRC).

The PointCheck™ device detected severe neutropenia with 92% sensitivity and 83% specificity in 24 subjects (Fig. 1-C). Two out of 26 subjects were flagged and excluded by the device Quality Control system.

With respect to usability, we found that 80.8% of participants scored above 80.8 on the SUS scale across all sites, with a mean SUS score of 89.1 across all sites (Fig. 1-D). The most common user errors seen were incorrect placement of the finger, hand, and arm.

Discussion

We have shown that PointCheck™, a novel technology for non-invasive, home-based neutropenia detection, is accurate and easy to use by first-time users in a simulated home environment with a mean SUS score of 89.1, indicating above average perception of user experience and falling within the top 10% of systems as evaluated by the SUS scale. The technology accurately detects severe neutropenia in the cohort of patients presented here, which included liquid tumors, other cancer conditions, and healthy volunteers. These preliminary results show that PointCheck™ is a promising technology to aid in the detection of severe neutropenia in the home setting. These results need to be confirmed in a larger patient cohort with a final device design.

Figure 1

View largeDownload slide

Figure 1

View largeDownload slide

Close modal